Visible Imaging Research Articles

Exploring Ear Recognition: Image Fusion and Deep Learning with Thermal and Visible Images

Exploring Ear Recognition: Image Fusion and Deep Learning with Thermal and Visible Images

Image Fusion Method Based on Snake Visual Imaging Mechanism and PCNN

The process of image fusion is the process of enriching an image and improving the image’s quality, so as to facilitate the subsequent image processing and analysis. With the increasing importance of image fusion technology, the fusion of infrared and visible images has received extensive attention. In today’s deep learning environment, deep learning is widely used in the field of image fusion. However, in some applications, it is not possible to obtain a large amount of training data. Because some special organs of snakes can receive and process infrared information and visible information, the fusion method of infrared and visible light to simulate the visual mechanism of snakes came into being. Therefore, this paper takes into account the perspective of visual bionics to achieve image fusion; such methods do not need to obtain a significant amount of training data. However, most of the fusion methods for simulating snakes face the problem of unclear details, so this paper combines this method with a pulse coupled neural network (PCNN). By studying two receptive field models of retinal nerve cells, six dual-mode cell imaging mechanisms of rattlesnakes and their mathematical models and the PCNN model, an improved fusion method of infrared and visible images was proposed. For the proposed fusion method, eleven groups of source images were used, and three non-reference image quality evaluation indexes were compared with seven other fusion methods. The experimental results show that the improved algorithm proposed in this paper is better overall than the comparison method for the three evaluation indexes.

Light-weight Fusion Network for UAV Visible-light and Infrared Images based on Real-time Brain-like Intelligent Computing

Abstract. Multiple sensors equipped on the Unmanned Aerial Vehicle (UAV) enables the acquisition of multi-modal and multi-source remote sensing data. UAV remote sensing usually faces with real-time or near-real-time tasks in complex and highly dynamic environments, such as disaster monitoring, traffic management, border patrol and so on. Under these conditions, the image fusion algorithm needs to be high efficiency, precision and reliability. In this paper, we proposed an intelligent real-time fusion network for UAV multi-source remote sensing data based on AI brain-like chips, and deployed the algorithm on the UAV platform to achieve online high-efficiency computing. Firstly, we have developed a novel image fusion algorithm named SFNet for infrared and visible image fusion based on ShuffleNetv2. Then, we use ZCA and l1-norm to process the remodeled deep feature. The weight maps are generated by bi-cubic interpolation and soft-max operation. Finally, the fused image is reconstructed by weighted-average operation. The proposed SFNet is deployed on the Lynxi KA200 brain computing chip, and a comprehensive inference test is carried out with UAV remote sensing data. Several State-Of-The-Art (SOTA) data fusion algorithms are deployed on the same chip for experimental comparison. The proposed SFNet is proved to have faster inference speed and better feature extraction results on brain-like chips. It is more suitable for real-time UAV remote sensing image fusion tasks.

Recognition of Plastic Film in Terrain-Fragmented Areas Based on Drone Visible Light Images

In order to meet the growing demand for food and achieve food security development goals, contemporary agriculture increasingly depends on plastic coverings such as agricultural plastic films. The remote sensing-based identification of these plastic films has gradually become a necessary tool for agricultural production management and soil pollution prevention. Addressing the challenges posed by the complex terrain and fragmented land parcels in karst mountainous regions, as well as the frequent presence of cloudy and foggy weather conditions, the extraction efficacy of mulching films is compromised. This study utilized a DJI Mavic 2 Pro UAV to capture visible light images in an area with complex terrain features such as peaks and valleys. A plastic film sample dataset was constructed, and the U-Net deep learning model parameters integrated into ArcGIS Pro were continuously modified and optimized to achieve precise plastic film identification. The results are as follows: (1) Sample quantity significantly affects recognition performance. When the sample size is 800, the accuracy of plastic film extraction notably improves, with area accuracy reaching 91%, a patch quantity accuracy of 96.38%, and an IOU and F1-score of 85.89% and 94.20%, respectively, compared to the precision achieved with a sample size of 300; (2) Different learning rates, batch sizes, and iteration numbers have a certain impact on the training effectiveness of the U-Net model. The most suitable model parameters improved the training effectiveness, with the highest training accuracy achieved at a learning rate of 0.001, a batch size of 10, and 25 iterations; (3) Comparative experiments with the Support Vector Machine (SVM) model validate the suitability of U-Net model parameters and sample datasets for precise identification in rugged terrains with fragmented spatial distribution, particularly in karst mountainous regions. This underscores the applicability of the U-Net model in recognizing plastic film coverings in karst mountainous regions, offering valuable insights for agricultural environmental health assessment and green planting management in farmlands.

Observational signatures of charged rotating traversable wormhole: shadows and light rings with different accretions

In this work, we investigate the optical appearance of qualitatively new observational features of accretion disk images around the charged rotating traversable wormhole (TWH) space-time for different spin, throat, and charge values. To accomplish this, we first consider the Hamilton–Jacobi method to derive the geodesic equations for the motion of photons and study the effects of parameters on the photon orbit in the observer’s sky. We found that each parameter affects the size and shape of the wormhole (WH) shadow and flatness is observed in the shadow because of spin and other parameters. To produce shadow images of sufficient visual quality but within manageable computational times, we adopt the ray-trace procedure and characterize the significant features of light trajectories on the observer’s screen, depending on the interaction between the space-time structure and the accretion disk. In addition, we consider the static spherically symmetric accretion flow model to observe the specific intensity around the traversable WH space-time geometry. It is found that the intensity and positions of the photon ring vary with respect to the involved parameters. In future observation, this type of study may provide a fertile playground to test the nature of compact objects, specifically the WH in the strong-field regime.

3D Poissonian image deblurring via patch-based tensor logarithmic Schatten-p minimization

In medical and biological image processing, multi-dimensional images are often corrupted by blur and Poisson noise. In this paper, we first propose a new tensor logarithmic Schatten-p (t-log-Sp ) low-rank measure and a tensor iteratively reweighted Schatten-p minimization algorithm for minimizing such measure. Furthermore, we adopt this low-rank measure to regularize the non-local tensors formed by similar 3D image patches and develop a patch-based non-local low-rank model. The data fidelity term of the model characterizes the Poisson noise distribution and blur operator. The optimization model is further solved by an alternating minimization technique combined with variable splitting. Experimental results tested on 3D fluorescence microscope images show that the proposed patch-based tensor logarithmic Schatten-p minimization method outperforms state-of-the-art methods in terms of image evaluation metrics and visual quality.

MRASFusion: A multi-scale residual attention infrared and visible image fusion network based on semantic segmentation guidance

To address the challenges of inadequate preservation of prominent targets, poor retention of texture details, and unsatisfactory reconstruction of image backgrounds in image fusion. In this paper, a multi-scale residual attention fusion network based on semantic segmentation guidance is proposed, termed as MRASFusion. First of all, Swin Transformer segmentation mask with high precision, and strong scalability is adopted to avoid the inefficiency and error of manual segmentation mask. The mask generated by semantic segmentation is used to construct a loss function to guide the image fusion process. Secondly, in order to maintain the integrity of contextual information and texture details, a new feature extraction module is proposed to fully extract the meaningful features. Finally, the fused image is obtained by reconstructing the extracted features. To verify the effectiveness of the method, MRASFusion is qualitatively and quantitatively compared with nine state-of-the-art fusion methods on TNO and RoadScene datasets. Experimental results indicate that our method has demonstrated satisfactory performance in image fusion tasks, exhibiting superior capabilities in preserving target information and retaining texture details. Furthermore, our fusion results have brought some performance improvements for advanced vision tasks, i.e., improved accuracy for the object detection, which provides a better foundation for solving real-world problems.

Geographical and seasonal variations of gravity wave activities in the upper mesosphere measured by space-borne imaging of molecular oxygen nightglow

Geographical and seasonal variations of gravity wave events in the upper mesosphere were investigated using the nightglow imaging data obtained by the Visible and near-Infrared Spectral Imager (VISI) on the Ionosphere, Mesosphere, upper Atmosphere and Plasmasphere (IMAP) onboard the International Space Station (ISS). The nadir-imaging data of the O2(0–0) atmospheric band (762 nm) with the typical emission peak around 95 km altitude was used to investigate small-scale waves (horizontal wavelengths less than ~ 200 km) on a global scale. To detect gravity wave events, the variance of high-pass filtered nightglow images within a local 100 km radius was evaluated, with a threshold set at three times the standard deviation from the average variance of the background level. A data screening algorithm that evaluates the variance of upwelling contamination light emission was also introduced to remove contaminated data. Applying the variance filter and data screening algorithm to a nearly 3-year data set, from November 2012 to August 2015, occurrence maps of wave events for four seasons were derived. The occurrence maps show a higher frequency of wave events in winter high latitudes (> 40° N/S), considerably attributed to gravity wave activity associated with the polar night jet. Hot spots were observed near orographic sources in winter high latitudes, including the eastern part of North America, Europe, and the southern Andes. In the summer hemisphere, hot spots were detected at mid-to-high latitudes such as North America, Europe, and the eastern side of the Eurasian continent, and at equatorial latitudes just above the intertropical convection zone (ITCZ). They are likely gravity waves from deep convection that arise from mid-latitude summertime thunderstorms and the ITCZ, respectively. During the equinox seasons, hot spots were detected near convective sources such as the Amazon Rainforest, Congo Rainforest, and the Indochina peninsula.Graphical

Infrared and visible image fusion method based on visual saliency objects and fuzzy region attributes

Infrared and visible image fusion method based on visual saliency objects and fuzzy region attributes

Enhancing Power Lines Detection Using Deep Learning and Feature-Level Fusion of Infrared and Visible Light Images

AbstractThe detection of power lines is critical for flight safety, especially for drones and low-flying aircraft. Power line detection models help prevent collisions, reducing potential damage and preserving lives, while also safeguarding critical infrastructure. This has led to significant research efforts to develop accurate detection models. In this study, we employ paired infrared–visible power line datasets to train three distinct deep learning models. The first two models are sequential deep learning models based on VGG16 and AlexNet networks. They are tailored for detection in visible images, while they were optimized again for infrared images. For the third model, we introduce an innovative deep learning architecture utilizing Functional Application Programming Interface, affording us the flexibility to construct a multi-input model with shared layers. Our proposed model accepts paired images (visible and infrared) as inputs. Then, a feature-level fusion process is applied to merge the extracted features from both inputs and generate an enriched feature map. This approach amalgamates the advantages of visible images, which boast high resolution and rich texture features, with infrared images, which excel in high contrast and clear vision under adverse environmental conditions. Comparing the outcomes of the three models, our proposed model emerges as the front runner, boasting an impressive accuracy rate of 99.37%. Moreover, real-time processing was adopted by conducting ablation experiments to optimize the model and reduce the number of trainable parameters, resulting in an inference speed of 2.7 milliseconds per frame.

High-visibility Ghost Imaging with Phase-controlled Discrete Classical Light Sources

Abstract In this paper, we take phase modulation to create discrete phase-controlled sources and realize the superbunching effect by a phase-correlated method. From theoretical and numerical simulations, we find the space translation invariance of the bunching effect is a key point for the ghost imaging realization. Experimentally, we create the orderly phase-correlated discrete sources which can realize high-visibility second-order ghost imaging than the result with chaotic sources. Moreover, some factors affecting the visibility of ghost image are discussed in detail.

SCAKD: a knowledge distillation framework based on spatial-corner attention for infrared and visible image fusion

SCAKD: a knowledge distillation framework based on spatial-corner attention for infrared and visible image fusion

CABnet: A channel attention dual adversarial balancing network for multimodal image fusion

Infrared and visible image fusion aims to generate informative images by leveraging the distinctive strengths of infrared and visible modalities. These fused images play a crucial role in subsequent downstream tasks, including object detection, recognition, and segmentation. However, complementary information is often difficult to extract. Existing generative adversarial network-based methods generate fused images by modifying the distribution of source images' features to preserve instances and texture details in both infrared and visible images. Nevertheless, these approaches may result in a degradation of the fused image quality when the original image quality is low. Considering the balance of information from different modalities can improve the quality of the fused image. Hence, we introduce CABnet, a Channel Attention dual adversarial Balancing network. CABnet incorporates a channel attention mechanism to capture crucial channel features, thereby, enhancing complementary information. It also employs an adaptive factor to control the mixing distribution of infrared and visible images, which ensures the preservation of instances and texture details during the adversarial process. To enhance efficiency and reduce reliance on manual labeling, our training process adopts a semi-supervised learning strategy. Through qualitative and quantitative experiments across multiple datasets, CABnet surpasses existing state-of-the-art methods in fusion performance, notably achieving a 51.3% enhancement in signal to noise ratio and a 13.4% improvement in standard deviation.

Image fusion using Y-net-based extractor and global-local discriminator

Although some deep learning-based image fusion approaches have realized promising results, how to extract information-rich features from different source images while preserving them in the fused image with less distortions remains challenging issue that needs to be addressed. Here, we propose a well worked-out GAN-based scheme with multi-scale feature extractor and global-local discriminator for infrared and visible image fusion. We use Y-Net as the backbone architecture to design the generator network, and introduce the residual dense block (RDblock) to yield more realistic fused images for infrared and visible images by learning discriminative multi-scale representations that are closer to the essence of different modal images. During feature reconstruction, the cross-modality shortcuts with contextual attention (CMSCA) are employed to selectively aggregate features at different scales and different levels to construct information-rich fused images with better visual effect. To ameliorate the information content of the fused image, we not only constrain the structure and contrast information using structural similarity index, but also evaluate the intensity and gradient similarities at both feature and image levels. Two global-local discriminators that combine global GAN with PatchGAN as a unified architecture help to dig for finer differences between the generated image and reference images, which force the generator to learn both the local radiation information and pervasive global details in two source images. It is worth mentioning that image fusion is achieved during confrontation without fusion rules. Lots of assessment tests demonstrate that the reported fusion scheme achieves superior performance against state-of-the-art works in meaningful information preservation.

MINM: Marine Intelligent Netting Monitoring Using Multi-Scattering Model and Multi-Space Transformation

Marine intelligent net tank aquaculture monitoring plays an important role in improving aquaculture efficiency, environmental monitoring efficiency, and environmental safety. The underwater environment has complex light, often with problems such as scattering and absorption, resulting in poor image quality, making it difficult to accurately analyze and judge the aquaculture environment. Improving marine intelligent net tank aquaculture monitoring has the following three advantages: 1) better observation and monitoring of the aquaculture process, timely detection of problems and abnormalities, to protect the benefits of aquaculture and product quality. 2) more convenient and rapid monitoring of the aquaculture environment, improving monitoring efficiency and reducing monitoring costs. 3) effective monitoring of the underwater environment around the farm, and timely detection of foreign pollution, harmful substances, and other problems, to protect the safety of the aquaculture environment. Therefore, in order to solve the two degradation problems of scattering and absorption in the process of marine smart net farm monitoring, we propose a marine smart net farm monitoring method using multiple scattering models and multiple spatial transformations, called MINM. Specifically, inspired by the image chromatic aberration correction method, we design a color correction method in the multicolor space, which is implemented by using the Lab and RGB color space by performing contrast-constrained adaptive histogram equalization and gray world assumptions, respectively, to correct color shifts in different color models. Based on this, we propose a de-scattering method using a multi-scattering model, which eliminates the effect of scattering on underwater imaging by embedding a complete multi-scattering underwater imaging model to guide the extraction of different features in the multi-scattering model. To obtain more qualified results, we also propose an efficient perceptual fusion to mix the output of the de-scattering and color correction. Thus, our method can take advantage of multiple scattering models and multiple spatial transformations to effectively improve the visual quality of underwater images, producing enhanced results that fit the complete underwater imaging model and have bio-visual characteristics. In extensive experimental demonstrations, our MINM method has shown higher performance than the state-of-the-art methods in terms of both visual quality and quantitative metrics. All experimental results and datasets in this paper are available from the following website: https://github.com/An-Shunmin/MINM.

Conditional generative model with skip-connection structure for low-light image enhancement

Vision-guided coal mine robots often encounter challenges in low-light environments, capturing images marked by poor visibility and significant loss of detail, which complicates the advancement of smart mine safety production. Despite various advancements, achieving high visibility and detailed images through robotic vision sensors remains a formidable task. This paper introduces a conditional generative model incorporating a skip-connection structure to address these issues. We present a novel approach utilizing conditional Generative Adversarial Networks (GNAs) aimed at improving the visibility of image content. This involves an encoder-decoder framework with skip-connections functioning as the generator, and the discriminator, both tailored to enhance image detail. In addition, we implement loss functions integrated with boundary equilibrium constraints to counter the issue of model collapse in conditional GANs, thereby enhancing training stability and image fidelity. Our experimental findings demonstrate that our model delivers competitive results against other state-of-the-art low-light image enhancement models, alongside a standardized training regimen that ensures rapid and consistent convergence.

VCAFusion: An infrared and visible image fusion network with visual perception and cross-scale attention

Infrared and visible image fusion methods aim to combine salient target instances and abundant texture details into fused images. However, due to the interference of harsh conditions, such as dense smoke, fog, and intense light, it is feasible for external interference information to be integrated into the fused image, which seriously affects the image quality. To this end, we propose an infrared and visible image fusion network with visual perception and cross-scale attention modules, termed VCAFusion, which integrates critical information from source images in harsh conditions more efficiently. Specifically, considering that the human eye can identify key information under adverse conditions, we design a visual perception module (VPM), guiding information integration from the perspective of human visual perception. In addition, we propose a cross-scale attention module (CSAM) based on shifted window cross-attention, which aims to capture the long-distance correlation between adjacent scale features, providing more accurate image information for image restoration. Experiments with a variety of datasets reveal that the VCAFusion can adaptively retain image information and further improve image generation ability in harsh conditions.

Edge preserving infrared and visible image fusion with three layer decomposition based on multi-level co-occurrence filtering

By merging different portraits of a particular scene, image fusion attempts to create a blended image that combines details from all the images. Infrared (IR) and visible image fusion can be accomplished in a variety of ways, including recent deep-learning-based techniques. However, edge-preserving filter (EPF) based fusion works well since it retains all the information from both images. Local filtering-based techniques, on the other hand, limit the fusion performance by introducing multiple gradient reversal artifacts and halos. This work presents an advanced IR and visible image fusion approach depending on three-level decomposition using multi-level co-occurrence filtering, which aims to overcome the common shortfalls such as halo effects seen in existing EPF based fusion. The reference images are decomposed in to base layer, small-scale layers and large-scale layers using multi-level co-occurrence filtering (MLCoF). Since most of the low frequency details are contained in the base layer, the conventional merging strategy by averaging is replaced with novel foreground information map (FIM) based fusion strategy. Small-scale layers are combined by applying max-absolute fusion strategy. A novel weight-map guided edge preserving fusion strategy is put forward for the integration of large-scale layers. Later, fused image is generated by the superposition of these different layers. Subjective visual and objective quantitative analysis shows that the suggested technique attains more notable performance in contrast with other modern fusion methods including many deep-learning techniques. In terms of visual perspective view, the results produced by the proposed approach are superior and include all details from both images. Additionally, it produces outcomes free of gradient reversal and halo artifacts.

SDFuse: Semantic-injected dual-flow learning for infrared and visible image fusion

Infrared and visible image fusion (IVIF) strives to render fused results that preserve the strengths of the source images (e.g., texture details and thermal highlights) while boosting downstream tasks. However, existing semantic-driven methods typically combine segmentation and fusion tasks via cascade architecture, which fails to sufficiently exploit semantic knowledge. To address this limitation, an innovative semantic-injected network with dual flows is proposed for IVIF, called SDFuse. Concretely, we first apply two pseudo-Siamese feature extraction modules (FEM) to obtain shallow structure features, which involve skip connections to prevent the disappearance of information. Subsequently, dual flows are constructed to support the joint learning paradigm of two tasks. The segmentation flow excavates abundant semantic knowledge from shallow features through fulfilling high-level segmentation task. The fusion flow is introduced to achieve the aggregation of highlights and textures. Note that SGM in fusion flow effectuates the injection of deep semantic knowledge into shallow structural features. Eventually, the region consistency module (RCM) with group normalization leads the network to focus on foreground objects and generate context-consistent fusion images. Our fusion results not only effectively fuse the information of the source image but also have advantages in downstream tasks. Extensive experiments over multiple representative benchmarks prove that SDFuse outperforms current state-of-the-art approaches.

Near-real-time multiparametric seismic and visual monitoring of explosive activity at Sabancaya volcano, Peru

This study presents the development of a multiparametric system that utilizes artificial intelligence techniques to identify and analyze volcanic explosions in near real-time. The study analyzed 1343 explosions recorded between 2019 and 2021, along with seismic, meteorological, and visible image data from the Sabancaya volcano. Deep learning algorithms like the U-Net convolutional neural network were used to segment and measure volcanic plumes in images, while boosting-based machine learning ensembles were used to classify seismic events related to ash plumes. The findings demonstrate that these approaches effectively handle large amounts of data generated during seismic and eruptive crises. The U-Net network achieved precise segmentation of volcanic plumes with over 98% accuracy and the ability to generalize to new data. The CatBoost classifier achieved an average accuracy of 94.5% in classifying seismic events. These approaches enable the real-time estimation of eruptive parameters without human intervention, contributing to the development of early warning systems for volcanic hazards. In conclusion, this study highlights the feasibility of using seismic signals and images to detect and characterize volcanic explosions in near real-time, making a significant contribution to the field of volcanic monitoring.